Patient specific risk prediction of cardiac events from image-derived cardiac function features

ABSTRACT

Systems and methods for predicting a patient specific risk of cardiac events for cardiac arrhythmia are provided. A medical image sequence of a heart of a patient is received. Cardiac function features are extracted from the medical image sequence. Additional features are extracted from patient data of the patient. A patient specific risk of a cardiac event is predicted based on the extracted cardiac function features and the extracted additional features.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/889,195, filed Aug. 20, 2019, the disclosure of which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present invention relates generally to patient specific riskprediction of cardiac events, and in particular to patient specific riskprediction of cardiac events for cardiac arrhythmia from image-derivedcardiac function features and other multi-modal features.

BACKGROUND

Cardiac arrhythmia is a condition in which the heart rate of a patientis irregular. A heart rate that is too fast is referred to astachycardia while a heart rate that is too slow is referred to asbradycardia. While most types of cardiac arrhythmia do not pose seriousrisks, some cardiac arrhythmias may cause major implications, such asstroke, heart failure, or death. Accordingly, one important task ispredicting risk of cardiac events for patients exhibiting symptoms ofcardiac arrhythmia.

In current clinical practice, cardiac arrhythmia is typically treatedfollowing standard global rules, resulting in an overly broad selectionof patients treated invasively. For example, patients with atrialfibrillation (AF), the most common type of cardiac arrhythmia in humans,are typically treated with catheter ablation. However, catheter ablationis associated with a high recurrence rate after ablation due toinsufficient patient selection.

One commonly used tool for measuring the predictiveness of certainfeatures on events is the Cox proportional-hazards model. Usingunivariate and multivariate Cox models, certain cardiac functional andstructural features, such as the left atrium function and left atriumvolume, have been found to be associated with different types of cardiacarrhythmia. While such features have clear physical meaning, such Coxmodels do not take into account all of the underlying features thatexist in medical images and other clinical data that are useful as riskpredictors for future cardiac arrhythmias.

BRIEF SUMMARY OF THE INVENTION

In accordance with one or more embodiments, systems and methods forpredicting a patient specific risk of a cardiac event for cardiacarrhythmia are provided. A medical image sequence of a heart of apatient is received. Cardiac function features are extracted from themedical image sequence. Additional features are extracted from patientdata of the patient. A patient specific risk of a cardiac event ispredicted based on the extracted cardiac function features andoptionally the extracted additional features.

In one embodiment, the patient specific risk of the cardiac event may bepredicted by determining a risk score representing the patient specificrisk of the cardiac event. The patient specific risk of the cardiacevent may be classified based on the risk score.

In one embodiment, the cardiac function features are extracted from themedical image sequence by encoding pairs of images of the medical imagesequence into the cardiac function features using a machine learningbased feature extractor network. The additional features are extractedfrom the patient data of the patient by encoding the patient data of thepatient into additional features using one or more additional machinelearning based feature extractor networks. The patient specific risk ofthe cardiac event is predicted by concatenating the cardiac functionfeatures and the additional features to form a feature vector andencoding the feature vector to features representing the patientspecific risk of cardiac events using a machine learning based riskregression network. The machine learning based feature extractornetwork, the one or more additional machine learning based featureextractor networks, and the machine learning based risk regressionnetwork may be individually trained or trained together.

These and other advantages of the invention will be apparent to those ofordinary skill in the art by reference to the following detaileddescription and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a framework for predicting a patient specific risk of acardiac event for a patient, in accordance with one or more embodiments;

FIG. 2 shows a method for predicting a patient specific risk of acardiac event, in accordance with one or more embodiments;

FIG. 3 shows a framework for training a plurality of machine learningnetworks for predicting a patient specific risk of a cardiac event, inaccordance with one or more embodiments;

FIG. 4 shows a plurality of exemplary neural networks that may be usedto implement one or more feature extractor networks described herein, inaccordance with one or more embodiments; and

FIG. 5 shows a high-level block diagram of a computer.

DETAILED DESCRIPTION

The present invention generally relates to methods and systems forpatient specific risk prediction of a cardiac event from image-derivedcardiac function features. Embodiments of the present invention aredescribed herein to give a visual understanding of such methods andsystems. A digital image is often composed of digital representations ofone or more objects (or shapes). The digital representation of an objectis often described herein in terms of identifying and manipulating theobjects. Such manipulations are virtual manipulations accomplished inthe memory or other circuitry/hardware of a computer system.Accordingly, is to be understood that embodiments of the presentinvention may be performed within a computer system using data storedwithin the computer system.

Embodiments described herein utilize underlying features extracted frommedical imaging data and other multi-modal sources of patient data forpredicting a patient specific risk of a cardiac event. The patientspecific risk of the cardiac event may be for cardiac arrhythmia or anyother cardiac condition (e.g., heart failure). A cardiac event includesany medical event associated with the heart of a patient, such as, e.g.,outcomes of treatment associated with the heart, occurrence orrecurrence of a major adverse cardiovascular event (MACE), or any otherevent associated with the heart. Advantageously, the patient specificrisk of a cardiac event is predicted in accordance with embodimentsdescribed herein utilizing more features extracted from medical imagesand other patient or clinical data as compared to conventionalapproaches, resulting in a more accurate and improved patient specificprediction of risk of a cardiac event.

It should be understood that while embodiments described herein aredescribed with respect to the predicting risk of cardiac events forcardiac arrhythmias, such embodiments are not so limited. Embodimentsmay be applied for predicting risk of cardiac events for any diseaseassociated with the heart of a patient.

FIG. 1 shows a framework 100 for predicting a patient specific risk of acardiac event, in accordance with one or more embodiments. At step 102,a medical image sequence is acquired. The medical image sequence may bea cine-MRI (magnetic resonance imaging) image sequence or an imagesequence of any other suitable modality (e.g., ultrasound or computertomographic images). At step 104, cardiac function features are learnedfrom the image sequence. Exemplary cardiac function features arerepresented in framework 100 as features of the original image sequence,motion features, motion grid features, compensation features deformationfeatures, and Jacobian determination features. At step 106, a patientspecific risk of cardiac events is predicted using the cardiac functionfeatures and, optionally, other multi-modal features, such as, e.g.,electrophysiological (EP) data. At step 108, the patient specific riskof cardiac events is classified. The classification may comprise, forexample, a classification of risk associated with an invasive treatment,the risk associated with success of a treatment, and the risk ofrecurrence of a cardiac event.

FIG. 2 is a method 200 for predicting a patient specific risk of acardiac event, in accordance with one or more embodiments. Method 200 ofFIG. 2 will be described with continued reference to FIG. 1. The stepsof method 200 may be performed by one or more suitable computingdevices, such as computer 502 of FIG. 5.

At step 202, a medical image sequence of a heart of a patient isreceived. In one example, the medical image sequence may be the medicalimage sequence acquired at step 102 of FIG. 1. The medical imagesequence may be a time sequence of medical images (or frames) showingdynamic motion of the heart of the patient. In one embodiment, themedical image sequence is a cine-MRI image sequence. However, themedical image sequence may be acquired using any suitable imagingmodality. For example, the medical image sequence may be a sequence ofMR images, cine-MRI images, computed tomography (CT) images,echocardiogram images, x-ray images, or medical images acquired usingany other medical imaging modality or combinations of medical imagingmodalities. The medical image sequence may be a sequence of 2D medicalimages or 3D medical volumes. The medical image sequence may be receiveddirectly from an image acquisition device, such as an MR scanner, CTscanner, ultrasound scanner, etc., as the medical image sequence of thepatient is acquired, or can be received by loading a previously acquiredmedical image sequence of the patient from a storage or memory of acomputer system or receiving a medical image that has been transmittedfrom a remote computer system.

At step 204, cardiac function features are extracted from the medicalimage sequence. In one example, the cardiac function features are thecardiac function features learned at step 104 of FIG. 1. The cardiacfunction features are low dimensional features extracted from themedical image sequence that relate to cardiac function. The cardiacfunction features may include features of the original medical imagesequence, motion features, motion grid features, compensation features,deformation features, Jacobian determination features, or any othersuitable feature.

The cardiac function features may be extracted from pairs of images inthe medical image sequence using a trained machine learning basedfeature extractor. The feature extractor may be an encoder network, suchas the encoder networks shown and described with respect to FIGS. 3 and4. Image pairs of the medical image sequence are input into the encodernetwork and each image pair is separately encoded by the encoder networkinto respective cardiac function features. In one embodiment, each imagepair comprises moving image I₀ and a respective fixed image I_(t) of themedical image sequence having T+1 frames, where t∈[1, T+1]. The encodernetwork may be trained during a prior offline or training stage,together with a decoding network, as shown and described with respect toFIG. 3 below. In one embodiment, feature extractor is the encodernetwork of the motion model described in U.S. patent application Ser.No. 16/834,269, filed Mar. 30, 2020, the disclosure of which is hereinincorporated by reference in its entirety.

At step 206, optionally, additional features are extracted from patientdata of the patient. In one example, the additional features are theother multi-modal features used to predict risk at step 106 of FIG. 1.The additional features may be extracted from patient data of thepatient, such as, e.g., EP data (e.g., echocardiograms or invasiveanatomical maps), images, meshes, or clinical data (e.g., bloodanalytics or patient characteristics) of the patient. The additionalfeatures may be extracted from patient data using one or more featureextractors, such as, e.g., one or more of the feature extractors shownand described with respect to FIGS. 3 and 4. Such feature extractorsextract expressive low dimensional features from high-dimensionalpatient data. The feature extractors may be selected based on the typeof patient data from which the additional features are extracted. Thefeature extractors are trained during a prior offline or training stage,as shown and described with respect to FIG. 3 below

At step 208, a patient specific risk of a cardiac event is predictedbased on the extracted cardiac function features and, optionally, theextracted additional features and other low dimensional clinical data.The extracted cardiac function features, the extracted additionalfeatures, and the other low dimensional clinical data are concatenatedinto a feature vector and the feature vector is input into a machinelearning based risk regression network. In one embodiment, the riskregression network is an encoder network of a task-specific autoencoder,such as shown in FIG. 3. A risk regression network is trained for thecardiac event (e.g., atrial fibrillation recurrence, deadly arrhythmia,etc.) during a prior offline or training stage, together with a decodingnetwork, as shown and described with respect to FIG. 3 below.

The risk regression network encodes the feature vector into lowdimensional features representing a risk score for the cardiac event,such as, e.g., treatment outcomes, occurrence or recurrence, etc. In oneembodiment, the risk of cardiac events may be classified based on therisk score. For example, the classification may be a decision (e.g.,yes, no, or check later) to treat (e.g., invasively) the patient basedon the risk score, a level of risk (e.g., high, medium, or low)associated with treatment of the patient, a level of risk associatedwith occurrence or recurrence of a cardiac event (e.g., aftertreatment), or any other classification. The classification may bedetermined by comparing the risk score to one or more thresholds. Theprediction of cardiac arrhythmia may include the risk score and theclassification.

The risk score is defined by the logarithm of the hazard ratio astypical assumed in the Cox regression analysis. As described withrespect to FIG. 3 below, the risk regression network is trained usingthe negative log partial likelihood as the survival function overcensored training samples comprising future event data. To this end, therisk regression network represents a non-linear version of the Coxsurvival analysis.

In one embodiment, for example if the dimensionality of the patient datais not high, the patient data may be directly concatenated with theextracted cardiac function features and the other low dimensionalclinical data without performing step 206.

At step 210, the predicted patient specific risk of a cardiac event(e.g., the risk score and/or the classification) is output. For example,the predicted patient specific risk of a cardiac event can be output bydisplaying the predicted patient specific risk of a cardiac event on adisplay device of a computer system, storing the predicted patientspecific risk of a cardiac event on a memory or storage of a computersystem, or by transmitting the predicted patient specific risk of acardiac event to a remote computer system.

Advantageously, the patient specific risk of a cardiac event predictedin accordance with embodiments described herein is significantlyimproved by using task-specific cardiac function features and othermulti-modal features, as compared with conventional approaches that onlyuse a few manually extracted features. Embodiments described hereinutilize high-dimensional multi-modal patient data (e.g., EP data,images, etc.) to thereby use more features for risk prediction. Thecumbersome process of extracted hand-crafted features, such as leftatrium string is not required in accordance with embodiments describedherein. Such advantages of embodiments described herein are realized, inpart, by applying different neural networks trained as task-specificfeature extractors for extracting cardiac function features at step 204,extracting additional features at step 206, and predicting a patientspecific risk of cardiac events at step 208. Besides optional clinicalfeatures, only task-specific features (i.e., the cardiac functionfeatures and the additional features) are used for risk prediction, inaccordance with embodiments described herein.

In one use case, embodiments described herein may be implemented in acardiology system to provide a risk estimation of future cardiacarrhythmias to support physician decision-making for or against aninvasive treatment (e.g., AF ablation). A probabilistic motion model maybe learned to extract cardiac function features. The cardiac functionfeatures, and optionally additional features from other multi-modal datasources, are input into a non-linear risk regression model to predictthe risk of future cardiac arrhythmias. In an end-to-end training, allnetworks can be trained in a task-specific way such that features areoptimally suited for the risk estimation task.

FIG. 3 shows a framework 300 for training a plurality of machinelearning networks for predicting a patient specific risk of cardiacevents, in accordance with one or more embodiments. Framework 300comprises network 302 for extracting cardiac function features, one ormore networks 306 for extracting additional features, and network 304for predicting a patient specific risk of cardiac events. Networks 302,304, and 306 may be autoencoders each comprising an encoder network anda decoder network. In particular, network 302 comprises encoder network312 and decoder network 316. Networks 306 comprise encoder networks322-A, 322-B, . . . (collectively referred to as encoder networks 322)and decoder networks 326-A, 326-B, . . . (collectively referred to asdecoder networks 326). Network 304 comprises encoder network 330 anddecoder network 334.

Networks 302, 306 and 304 are trained during a prior offline or trainingstage using respective encoder networks 312, 322, and 330 and respectivedecoder networks 316, 326, and 334 according to framework 300. Oncetrained, networks 302, 306, and 304 are applied during an online orinference stage using respective encoder networks 312, 322, and 330. Forexample, encoder network 312 may be applied at step 204 of FIG. 2 toextract the cardiac function features, encoder networks 322 may beapplied at step 206 of FIG. 2 to extract the additional features, andencoder network 330 may be applied at step 208 of FIG. 2 to determine apatient specific prediction of cardiac arrhythmia. Decoder networks 316,326, and 334 are only used during the training stage in order toconstrain and regularize respective encoder networks 312, 322, and 330to avoid-overfitting and are not used during the inference stage.

Network 302 comprises encoder network 312 and decoder network 316.Encoder network 312 receives training image pairs 310-A, 310-B, . . . ,310-T (collectively referred to as training image pairs 310) of trainingmedical image sequence 308 having T+1 frames. Each training image pair310 comprises moving image I₀ and a respective fixed image I_(t) of thetraining medical image sequence, where t∈[1, T+1]. Encoder network 312independently encodes each training image pairs 310 into respectivecardiac function features z₀ 314-A, z₁ 314-B, . . . , z_(T) 314-T(collectively referred to as cardiac function features z_(t) 314),collectively forming function matrix z∈R^(DxT). Cardiac functionfeatures 314 are low dimensional features extracted from training imagepairs 310 that relate to cardiac function. Exemplary features includefeatures of the original images, motion features, motion grid features,compensation features, deformation features, Jacobian determinationfeatures, or any other suitable feature. Decoder network 316 determinesrespective deformation fields ϕ₀ 318-A, ϕ₁ 318-B, . . . , ϕ₇ 318-T(collectively referred to as deformation fields ϕ_(t) 314) from cardiacfunction features z_(t) 314. Deformation fields ϕ_(t) 318 representmotion between the training image pairs 310 and may be diffeomorphic.Deformation fields ϕ_(t) 318 may be applied by decoder network 316 totransform moving image I₀ to reconstruct respective fixed images I_(t).Accordingly, network 302 is trained to perform a temporal registrationof the moving image I₀ with each fixed image I_(t). Network 302 may betrained using any suitable loss function.

Networks 306 include a network for each type of training patient datafrom which additional features are to be extracted. The architecture ofeach of the networks 306 is based on the type of the patient data. Suchtraining patient data may comprise EP data 320-A (e.g., echocardiogramsinvasive-anatomical maps) and images 320-B (collectively referred to astraining patient data 320). Training patient data 320 may additionallyor alternatively include any other type of patient data, such as, e.g.,meshes, blood analytics, patient characteristics, etc. As shown inframework 300, encoder networks 322-A and 322-B respectively encode EPdata 320-A and images 320-B into low dimensional features 324-A and324-B (collectively referred to as additional features 324). Decodernetworks 326-A and 326-B respectively reconstruct EP data 320-A andimages 320-B from features 324-A and 324-B. Networks 306 may be trainedusing any suitable reconstruction loss function, such as, e.g.,mean-of-squared differences between the input and the output with aregularizer on the distribution of the features. Any other suitablereconstruction loss function may also be applied, such as, e.g.,reconstruction loss functions typically utilized for trainingautoencoders, denoising autoencoders, variational autoencoders, etc.FIG. 4 describes various architectures and loss functions on whichnetworks 306 may be configured and trained, in accordance with oneembodiment.

Network 304 comprises encoder network 330 and decoder network 334.Network 304 may be a non-linear risk regression model formed by atask-specific autoencoder, which learns an optimized latentrepresentation for risk prediction based on observed cardiac events.Network 304 is trained for a particular cardiac event using trainingdata for the particular cardiac event, and may be retrained for othercardiac events. The risk regression model may comprise multiple densenetwork layers. Encoder network 330 encodes feature vector x 328 intolow dimensional features 332. Feature vector x 328 comprises aconcatenation of cardiac function features 314, additional features 324,and other training clinical features. Decoder network 334 decodesfeatures 332 to reconstruct feature vector x 328 as reconstructedfeature vector x′ 336 as. Features 332 represent a patient specific riskscore 340 of cardiac events. The risk of cardiac events may beclassified based on risk score 340 using one or more thresholds. Forexample, the classification may include a decision for treatment (e.g.,yes, no, check later), the level of risk (e.g., high, medium, or low)associated with treatment, the level of risk of occurrence or recurrenceof a cardiac event, or any other classification. The classification maybe determined by comparing the risk score to one or more thresholds.

In one embodiment, network 304 is trained with loss function

using a feature reconstruction loss term

_(rec) combined with a supervised Cox survival loss term

_(risk), as defined in Equation (1):

=

_(rec)+γ

_(risk)  Equation (1)

where feature reconstruction loss term

_(rec) and supervised Cox survival loss term

_(risk) are respectively defined in Equations (2) and (3):

$\begin{matrix}{\mathcal{L}_{rec} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{{x_{i} - {p_{\omega}\left( {q_{\theta}\left( x_{i} \right)} \right)}}}^{2}}}} & {{Equation}\mspace{14mu} (2)} \\{\mathcal{L}_{risk} = {- {\sum\limits_{i = 1}^{N}{\delta_{i}\left\lbrack {{q_{\theta}\left( x_{i} \right)} - {\log \; {\sum\limits_{j = 1}^{N}{R_{ij}{\exp \left( {q_{\theta}\left( x_{i} \right)} \right)}}}}} \right\rbrack}}}} & {{Equation}\mspace{14mu} (3)}\end{matrix}$

Using censoring indicator δ_(i)=1 if event and δ_(i)=0 if censored, riskmatrix R_(ij)=1 if t_(j)≥t_(i) and R_(ij)=0 if t_(j)<t_(i), based on Nsamples per batch. This represents a non-linear Cox proportional hazardmodel where the risk score obtained from the r=q_(θ)(x_(i)) network 304is defined by the logarithm of the hazard ratio as typically assumed inthe Cox regression analysis.

In one embodiment, network 304 is trained for a plurality of cardiacevents to predict a general risk score for the plurality of cardiacevents.

In accordance with one embodiment, networks 302, 304, and 306 areindividually trained using training data comprising training imagesequences and other patient and/or clinical data for a patient cohortwith known future cardiac events. First, network 302 is trained usingthe training data in an unsupervised manner to obtain cardiac functionfeatures and networks 306 are trained using the training data to obtainthe additional features. Networks 302 and 306 may be trainedsequentially in any order or in parallel. Next, network 304 is trainedto extract features representing the risk score based on observedoutcomes in the training data using the extracted cardiac functionfeatures extracted by the trained network 302 and the additionalfeatures extracted by the trained network 306. One or more thresholdsare determined based on risk scores in the training data to classify therisk.

In accordance with another embodiment, networks 302, 304, and 306 arecollectively trained together in an end-to-end manner. In end-to-endtraining, network 304 is trained according to Equation (1) by extendingloss function

by weighted summands of all feature extract loss terms from networks 302and 206. One advantage of end-to-end training of networks 302, 304, and306 is that networks 302 and 306 are trained to extract task-specificfeatures that are optimized for risk prediction. However, end-to-endtraining comes with higher training costs. In particular, networks 302and 306 are expensive to train due to the high dimensionality of theinput data, which may be images or image sequences (with potentiallymillions of parameters). According, training networks 302 and 306 maytake a significant amount of time (e.g., up to 24 hours). Network 304may be trained more efficiently (e.g., a few minutes) since the inputfeatures have relatively low dimensionality (e.g., a few hundred orthousand input parameters).

In accordance with another embodiment, networks 302, 304, and 306 aretrained using a combination of individual training and end-to-endtraining by pre-training some of networks 302 and/or 306, while trainingthe remaining networks in an end-to-end manner with network 304.

FIG. 4 shows a plurality of exemplary neural networks 400 that may beused to implement one or more of the feature extractor networksdescribed herein, in accordance with one or more embodiments. Neuralnetworks 400 may be applied to learn cardiac function features at step104 of FIG. 1 or extract multi-modal features used at step 106 of FIG.1, applied to extract cardiac function features at step 204 of FIG. 2 orextract additional features at step 206 of FIG. 2, or may be networks302 or 306 of FIG. 3. One or more of the plurality of neural networks400 may be used based on the type of data from which features are to beextracted. As shown in FIG. 4, the plurality of neural networks comprisean idea feature extractor 402, a hand-crafted feature extractor 404trained with loss function Σ_(i)∥ƒ_(i)−h_(i)∥ using hand-craftedfeatures h_(i), a standard autoencoder 406 trained with loss function∥data−data′∥, a denoising autoencoder 408 trained with loss function∥data_(n)−data′∥ using noisy data data_(n), and a variationalautoencoder 410 trained with loss function∥data−data′∥+Σ_(i)KL(p(ƒ_(i)|data)∥p(ƒ_(i))) using prior distribution offeatures p(ƒ_(i)).

Systems, apparatuses, and methods described herein may be implementedusing digital circuitry, or using one or more computers using well-knowncomputer processors, memory units, storage devices, computer software,and other components. Typically, a computer includes a processor forexecuting instructions and one or more memories for storing instructionsand data. A computer may also include, or be coupled to, one or moremass storage devices, such as one or more magnetic disks, internal harddisks and removable disks, magneto-optical disks, optical disks, etc.

Systems, apparatus, and methods described herein may be implementedusing computers operating in a client-server relationship. Typically, insuch a system, the client computers are located remotely from the servercomputer and interact via a network. The client-server relationship maybe defined and controlled by computer programs running on the respectiveclient and server computers.

Systems, apparatus, and methods described herein may be implementedwithin a network-based cloud computing system. In such a network-basedcloud computing system, a server or another processor that is connectedto a network communicates with one or more client computers via anetwork. A client computer may communicate with the server via a networkbrowser application residing and operating on the client computer, forexample. A client computer may store data on the server and access thedata via the network. A client computer may transmit requests for data,or requests for online services, to the server via the network. Theserver may perform requested services and provide data to the clientcomputer(s). The server may also transmit data adapted to cause a clientcomputer to perform a specified function, e.g., to perform acalculation, to display specified data on a screen, etc. For example,the server may transmit a request adapted to cause a client computer toperform one or more of the steps or functions of the methods andworkflows described herein, including one or more of the steps orfunctions of FIGS. 1-2. Certain steps or functions of the methods andworkflows described herein, including one or more of the steps orfunctions of FIGS. 1-2, may be performed by a server or by anotherprocessor in a network-based cloud-computing system. Certain steps orfunctions of the methods and workflows described herein, including oneor more of the steps of FIGS. 1-2, may be performed by a client computerin a network-based cloud computing system. The steps or functions of themethods and workflows described herein, including one or more of thesteps of FIGS. 1-2, may be performed by a server and/or by a clientcomputer in a network-based cloud computing system, in any combination.

Systems, apparatus, and methods described herein may be implementedusing a computer program product tangibly embodied in an informationcarrier, e.g., in a non-transitory machine-readable storage device, forexecution by a programmable processor; and the method and workflow stepsdescribed herein, including one or more of the steps or functions ofFIGS. 1-2, may be implemented using one or more computer programs thatare executable by such a processor. A computer program is a set ofcomputer program instructions that can be used, directly or indirectly,in a computer to perform a certain activity or bring about a certainresult. A computer program can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment.

A high-level block diagram of an example computer 502 that may be usedto implement systems, apparatus, and methods described herein isdepicted in FIG. 5. Computer 502 includes a processor 504 operativelycoupled to a data storage device 512 and a memory 510. Processor 504controls the overall operation of computer 502 by executing computerprogram instructions that define such operations. The computer programinstructions may be stored in data storage device 512, or other computerreadable medium, and loaded into memory 510 when execution of thecomputer program instructions is desired. Thus, the method and workflowsteps or functions of FIGS. 2-4 can be defined by the computer programinstructions stored in memory 510 and/or data storage device 512 andcontrolled by processor 504 executing the computer program instructions.For example, the computer program instructions can be implemented ascomputer executable code programmed by one skilled in the art to performthe method and workflow steps or functions of FIGS. 2-4. Accordingly, byexecuting the computer program instructions, the processor 504 executesthe method and workflow steps or functions of FIGS. 2-4. Computer 502may also include one or more network interfaces 506 for communicatingwith other devices via a network. Computer 502 may also include one ormore input/output devices 508 that enable user interaction with computer502 (e.g., display, keyboard, mouse, speakers, buttons, etc.).

Processor 504 may include both general and special purposemicroprocessors, and may be the sole processor or one of multipleprocessors of computer 502. Processor 504 may include one or morecentral processing units (CPUs), for example. Processor 504, datastorage device 512, and/or memory 510 may include, be supplemented by,or incorporated in, one or more application-specific integrated circuits(ASICs) and/or one or more field programmable gate arrays (FPGAs).

Data storage device 512 and memory 510 each include a tangiblenon-transitory computer readable storage medium. Data storage device512, and memory 510, may each include high-speed random access memory,such as dynamic random access memory (DRAM), static random access memory(SRAM), double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devicessuch as internal hard disks and removable disks, magneto-optical diskstorage devices, optical disk storage devices, flash memory devices,semiconductor memory devices, such as erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), compact disc read-only memory (CD-ROM), digital versatile discread-only memory (DVD-ROM) disks, or other non-volatile solid statestorage devices.

Input/output devices 508 may include peripherals, such as a printer,scanner, display screen, etc. For example, input/output devices 508 mayinclude a display device such as a cathode ray tube (CRT) or liquidcrystal display (LCD) monitor for displaying information to the user, akeyboard, and a pointing device such as a mouse or a trackball by whichthe user can provide input to computer 502.

An image acquisition device 514 can be connected to the computer 502 toinput image data (e.g., medical images) to the computer 502. It ispossible to implement the image acquisition device 514 and the computer502 as one device. It is also possible that the image acquisition device514 and the computer 502 communicate wirelessly through a network. In apossible embodiment, the computer 502 can be located remotely withrespect to the image acquisition device 514.

Any or all of the systems and apparatus discussed herein, includingnetworks 302, 304, and 306, may be implemented using one or morecomputers such as computer 502.

One skilled in the art will recognize that an implementation of anactual computer or computer system may have other structures and maycontain other components as well, and that FIG. 5 is a high levelrepresentation of some of the components of such a computer forillustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the present invention and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention. Those skilled inthe art could implement various other feature combinations withoutdeparting from the scope and spirit of the invention.

1. A method comprising: receiving a medical image sequence of a heart ofa patient; extracting cardiac function features from the medical imagesequence; and predicting a patient specific risk of a cardiac eventbased on the extracted cardiac function features.
 2. The method of claim1, wherein predicting a patient specific risk of a cardiac event basedon the extracted cardiac function features comprises: determining a riskscore representing the patient specific risk of the cardiac event. 3.The method of claim 2, wherein predicting a patient specific risk of acardiac event based on the extracted cardiac function featurescomprises: classifying the patient specific risk of the cardiac eventbased on the risk score.
 4. The method of claim 1, further comprisingextracting additional features from patient data of the patient, whereinpredicting a patient specific risk of a cardiac event based on theextracted cardiac function features comprises: predicting the patientspecific risk of the cardiac event based on the extracted cardiacfunction features and the extracted additional features.
 5. The methodof claim 1, wherein extracting cardiac function features from themedical image sequence comprises: encoding pairs of images of themedical image sequence into the cardiac function features using amachine learning based feature extractor network.
 6. The method of claim5, further comprising: encoding patient data of the patient intoadditional features using one or more additional machine learning basedfeature extractor networks.
 7. The method of claim 6, wherein predictinga patient specific risk of a cardiac event based on the extractedcardiac function features comprises: concatenating the cardiac functionfeatures and the additional features to form a feature vector; andencoding the feature vector to features representing the patientspecific risk of the cardiac event using a machine learning based riskregression network.
 8. The method of claim 7, further comprising:individually training the machine learning based feature extractornetwork, the one or more additional machine learning based featureextractor networks, and the machine learning based risk regressionnetwork.
 9. The method of claim 7, further comprising: training themachine learning based feature extractor network, the one or moreadditional machine learning based feature extractor networks, and themachine learning based risk regression network together.
 10. Anapparatus comprising: means for receiving a medical image sequence of aheart of a patient; means for extracting cardiac function features fromthe medical image sequence; and means for predicting a patient specificrisk of a cardiac event based on the extracted cardiac functionfeatures.
 11. The apparatus of claim 10, wherein the means forpredicting a patient specific risk of a cardiac event based on theextracted cardiac function features comprises: means for determining arisk score representing the patient specific risk of the cardiac event.12. The apparatus of claim 11, wherein the means for predicting apatient specific risk of a cardiac event based on the extracted cardiacfunction features comprises: means for classifying the patient specificrisk of cardiac events based on the risk score.
 13. The apparatus ofclaim 10, further comprising means for extracting additional featuresfrom patient data of the patient, wherein the means for predicting apatient specific risk of a cardiac event based on the extracted cardiacfunction features comprises: means for predicting the patient specificrisk of the cardiac event based on the extracted cardiac functionfeatures and the extracted additional features.
 14. A non-transitorycomputer readable medium storing computer program instructions, thecomputer program instructions when executed by a processor cause theprocessor to perform operations comprising: receiving a medical imagesequence of a heart of a patient; extracting cardiac function featuresfrom the medical image sequence; and predicting a patient specific riskof a cardiac event based on the extracted cardiac function features. 15.The non-transitory computer readable medium of claim 14, whereinpredicting a patient specific risk of a cardiac event based on theextracted cardiac function features comprises: determining a risk scorerepresenting the patient specific risk of the cardiac event.
 16. Thenon-transitory computer readable medium of claim 14, wherein extractingcardiac function features from the medical image sequence comprises:encoding pairs of images of the medical image sequence into the cardiacfunction features using a machine learning based feature extractornetwork.
 17. The non-transitory computer readable medium of claim 16,further comprising: encoding patient data of the patient into additionalfeatures using one or more additional machine learning based featureextractor networks.
 18. The non-transitory computer readable medium ofclaim 17, wherein predicting a patient specific risk of a cardiac eventbased on the extracted cardiac function features comprises:concatenating the cardiac function features and the additional featuresto form a feature vector; and encoding the feature vector to featuresrepresenting the patient specific risk of the cardiac event using amachine learning based risk regression network.
 19. The non-transitorycomputer readable medium of claim 18, further comprising: individuallytraining the machine learning based feature extractor network, the oneor more additional machine learning based feature extractor networks,and the machine learning based risk regression network.
 20. Thenon-transitory computer readable medium of claim 18, further comprising:training the machine learning based feature extractor network, the oneor more additional machine learning based feature extractor networks,and the machine learning based risk regression network together.